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Background and purpose 

hu-rc^ising attention began to be paid to.pi'oblems of iini versity 
peLlagogies during the second half of the li)()Os. In Sweden a Com- 
mission (TPT) was appointed on the subjc^ct, and following in the 
\\:\kv of this Commission, ('valuation has developed during the pre-^ 
sent dcavnle into a specific fiehl of interest in high^er education. . ■ 
Th(* rising netul for evaluation can he attributed to a series of 
c^hanges oT'curring in recc^nt years. Rising numbers of students 
have gene rat chI hcMvy expenditure j^i the post -secondary sector, 
with the result that planning and pfficKMicy measures have become 
a f)rogressiv(*lv more u regent necessity. Evaluation has been under- 
taken to see v^hether the resources applied and the measures taken 
have yielded the return intendeci. Varu)us decisions in the field of 
educat:igi| policy, such as the expansion of upper secondary educa- 
tion and neu rules (^f admissu>n to higher cniucation, have, opened 
Uluvc^rsitv educatum oppoi-tunities to categories \s hich u e rc* foi'mer- 
Iv excluded. Student ^ roups have bet-ome moi-e heterogeneous, both 
soeioeconomieall V speaking and iti terms of stud\- aptitude and 
edtif iMonal background. Neu educ-atioiial variants, suc-li as univer- 
se ^ ' (j:. '^es eombint^d uith courses at upper secondary or voca- 
. moIs, a«d svstematic decentralized universitv studies, 
ha . b( < n int roducd 1 . 

I 

' At the same tune as inc^reasing demands have been made (or foliou- 
ilD and evaluation, r-ntieism has been levelled ygainst current 
methods of educatif)na T evaluati nn . Kducational statistics in the 
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' " form of examinatioQ result's, sJkudy p>olQts and degrees have been 
taken As the starting point for recommendations and decisions. 
Quantitative evaluation of this kind says little about the manner in 
which instruction is provided»and tbe changes, which should be ef- 
fected in order t6 raise its quality. EvalUatioa methods need to be 
developed which focus more upon the'Iatter aspects. ^ 

Our project, which began in 1972, is to be seen in this light. The 
superordinate aim has been, by means of a review of literature 
and the conduct of empirical studies, to eV6cidate and investigate , 
aspects which may have an essential bearing upon the evaluation 
of university instruction. Various mutually complementary investi- 
gations have been undertaken to this end. 



esign and focus of the project 

The development of the project and^theoretical and empirical con- 
tributions made to it have been best described in a joint thesis en- 
titled "Evaluation of teaching - an analysis of problems together 
with some empirical studies at university level" (Franke-Wikberg 
if Johansson, 1975). The following summary is constructed on 
much the same lines as the thesis. ^ 



Analysis of literature * s 

Our analysis includes a review- of the meaning attached to the evalu- 
ation concept and to the general purpose and constituent activities 
of evaluations. Diffferent lines of development in' evaluation are dis- 
cussed as well as the^ relation of evaltratibn to educational research. 
Som^ examples of W*iluation models are p>esented, ^nd an appraisal 
is undertaken of their usefulness and function in the evaluation con- 
text. Different approach^^o evaluation are analysed in terms of 
their relation to educational and social attitudes. Evaluation prob- ' 
lems in evaluation are considered and there is a discussion of the 
data which should be collected and the methods which can be used. 
The analysis of literature provides a foundation for the empirical 
studies and leads on to an empirical model for evaluation. This 
model does not exist in vacao : we assume that instruction is ref- 
lated to a wider context. 



Goals and guidel(!^es 
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Figure 1. Model for evaluation wrk. 



BrtelR^we have regarded evaluation as a "comparison between 
observations and intentions of instruction''. The broken <lines in . 
the model indicate the discussion which can be held concerning 
the relation of the observations to the sUperordinate intentioni-of 
.teaching / . • i 

The model implies an investigatory approach involving a study b0th 
of eoals dnd guidelines and of the way in which they are apprehend- 
ed and concretized. The characteristics of the student group, the 
teacher- s aims in the planning of instruction, the content and or- 
ganization of the instruction and environmental factors are all ana- 
lysed. These very immediate preconditions of the teaching process, 
we term background infortnation . A study of the instruction should 
comprise the teacher and student components and the conterit.and • 
design of the instruction. Observations of the teaching process and 
th0 follow-u^of private study ajre important elements of evaluation. 
Situation-specific outcomes (process outcomes) are necessary, and 
different types of final outcome are of interest. The evaluator 
should try to relate these aspects to each other and view the instruc 
tion in its context. Efforts to shed light oh the teaching situation as 
a whole make it possible to compare the teachibg with its intentions 

The empiriqal studies which have been carried out can be divided 
into three blocks: • 

I Evaluations of courses in three different university subjects. 
These evaluations are directly linked to the overall view 
iAijJlied by the model, i.e. data have been collected before, 
during and after each course. 

•n Three surveys of previous knowledge in university subjects, . 
constituting deeper studies of a component of the model, r 

ni An investigation concerning factors influencing satisfaction 
and study achievement. Factors of tliis land have been ident- 
ified by means of a critical incident techni.que. This investi- 
gation reflects the model as a whole in the same way as the 
studies in point I, but ft has a more extensive character 
than the latter. 

Detailed investigations conducted 

/ Evaluation of courses in three different university subjects 
During the autumn term 1972, the course entitled "Analysis I" 
(9 points) was evaliiated. Evaluations during the spring term of- 
1973 concerned the ''Analysis H" (8 points) and "Theory of prob- 
ability and statistics" (7 points) mathematics courses. English 
studies have been evaluated with reference to the "Command ef 
written language" course (4 points) and history with reference to 
the "Modern histo'ry" (4 points) and "Ecpnomic history since 1914" 
(3 points) courses . 
■\ 

On the basis of an analysis of the superordinate intentions of 
teaching, expectations have been defined concerning the coriteat-^ 
and design of teaching and of the roles of teacher and students. 
' Certain central concepts have been identified. The analysis pro- 



vided the 'foundation for the selection of methods* and the collection 
of data.' 

Backgrourid information on fhie students has been efbllected via a 
questionnaire, tests of previous, know I edge and parts of an ability 
test. The questions included in the;questionnaire primarily concern 
factors which can be influenced or which can be directly utilized for 
teaching purposes at the department. Information concerning 1 a, 
c - e in the model has been obtained by interviews of teachers, 
inquiries to directors of studies, the study of course instructions 
iand timetables and direct observation. 

The teachers have defined goal instructions in the manner consider- 
ed most suitable for the subject in question. Students replying to the 
questionnaire have indicated the usefulness and use.pf the instruc- 
tions and expressed viewpoints concerning them. In English and his- 
tory, efforts have also been made to verbalize affective aims, and 
the fulfilment of these aims ha^ then been rated by the student^. 

A modified Flanders checklist has been devised for the observation 
of teaching. Instead of regarding the students as a unit we have 
chosen to 'study the role of each student in the process! Each indi-r 
vidual has therefore been identified. Observations have also been 
recorded in the farm of a diary. Questionnaires and interviews have 
yielded supplementary information concerning the instruction. 

A time estimation schedule has been produced to shed light on pri- 
vate study. One version has been made to include other activities 
besides those directly concerned with studies. Mathematics and , 
English students have made daily entries in the schedule for a week 
in the middle of the sub-course, and history students also made such 
entries during the week preceding'an examination. 

Examination results have been collected as a measure of the cogni-, 
tive outcome of the teaching processes. Student attitudes have been 
studied through fairly comprehensive questionnaires and, in the 
case of history students, via interview questions. Various types of 
questioa have been tested in the questionnaires. Certain teacher 
attitudes have transpired from the interviews. 

// Surveys of previous knowledge in university subjects 
In the first survey,^ an exploration was made of the incidfence of tests 
of previous knowledge at faculties of arts and sciences, together 
with their purposes, construction and Use. The second study de- 
scribes the previous knowledge which a number of university 
teachers expect of their students and their rating of the actual level 
of previous knowledge in the subject. Comparisons 'are made with 
the students' actual test perfomance. In the third study a technique 
for the revision of tests of previous knowledge is trifed out and their 
•relation. to teaching is examined. The second and the third study are 
both focussed on the subjects mathematics, English and history. 
These surveys ate described in the thesis* and they have also been 
separate^ reported (Educational Reports nos. 43, 44, University 
of Umei - published i)i Swedish;-^ summaries in English) . 



/// Factors influencing satisfaction and study achievement - 

identification usinj^tl^ej^h'tica/ incident technique 
About 1,200 concrete eveatsTjelieved to have had a positive or nega- 
tive effect on studies have been not.ed by 368 students. These eVents 
have been classified in accordance with the model, and the main 
areas have then ^edln subdivided into various sub-categories. The 
different categories have then been rated conceming the degree and 
direction of their influence on studies. The last mentioned of these 
ratings was carried out during the spring term of 1974 by mathemat-^ 
ics, English and history students at varidus universities in Sweden. 

Some findings ' \ 

The findings show that there is a great deal of variation in different 
. respects between students taking the same*subject. Previous knowl- 
edge, expectations and opinions conceming the way in which teach- 
ing should be oi^ganized all vary a great deal from one student to 
another. T^sts of previous knowledge are comparatively rare, and 
it is hard to discern any uniform* approach to their construction and 
use. Teachers have di££iculty in adapting their instruction to the 
students' previous knowledge. The demands made by teachers are 
^ top high in relation to the actual level of the students' knowledge. * 
Teaching is teacher-dominated and its content and form are fairly 
rigidly planned. Student activity is: relatively small during lessons. 
There is a great shortage of- time and studies focus on examinations 
to a striking ektent. The pattern of findings froAi the more extensive 
study (IE) agrees well with the pattern yielded by the more intensive * 
one (I) . 'Among olher things it has been found that the teacher is con- 
sidered to play sla. important part in determining both study achieve- 
ment and satisfaction and that co-operation between students, motiv- 
ation and study habits are rat e^H^^y^ by many respondents. Teacher 
viewpoints include the sentiirf^nt thai they do not have time to create 
a pleasant atmosphere in the teaching situation. Reference is also 
made to the lack of personal Qontact with students (III) . 

Our general impression in the light of the findings obtained from 
these studies is that certain superordinate intentions such as, for 
example, individualized instruction "and student activity are not 
catered for in practice. In the thesis an attempt is made to explain 
the image of ^teaching which we have' acquired. The possibility is 
discussed of transforming instruction in a manner conducive to the 
fulfilment of the above mentioned intentions. 

■ ' ' ■ • ■ - 

Some final impressions froi^ the work of the project 

One of the guidelines of the project has been to combine a research- 
oriented approach with a working method making it possible at the 
same time to participate in the continuous development work of the. 
departments. One result of this combination has been thatjii^ collec- 
tion aind analysis of data have been undertaken for different pur- 
poses ► Apart from the processing of data being both tiiri^-consuming 
*and conductl^d at a. level characteristic of research analysis, great 
importance -has also been attached to informing teacher^ and stu- 
dents as rafpidly as possible of the results of evaluation. Our close 
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contact with teachers and students shows among other things that 
we do not regard evaluation as a mechanical activity divorced from 
the business of teaching. ^ 

Apart from the fact that the above me^tlones sutvey strategies have 
had encouraging results, we mai|Valeo mention the advantages of 
various kinds accruing from the use bf a total evaluation model. 
These Include the following: 

— - ^ts approach has simplified the collection, Interpretation 
and analysis of data. i 

< < ; ■ \ 

- It has opened the w^y to attempts to explalij the current 
teaching situation. ' ^ ' * r c - - 

- We have been able to carry out various subsidiary Investiga- 
tions within the same theoretical framework, which has niade 
It possible for our studies to gain In depth as well as width. 

- Our emphasis on process data and the detailed analysis we 

■ ' have conducted at Individual and teaching group levels have 
f yielded valuabl«^nfbrmatlon which can be uSed for didactic ^ 
purposes and In the developnient of theory. 

- Closer studies, e.g. concerning previous knowledge, have been 
capable of leading to pedagoglcally enlightening alignments. 

- Attention has been given to other measures of outcome besides 
study achievement . „ 

The thesis also discussed when, how and by whom evaluation Is to 
be performed and what use can be made of evaluation results. Sug- 
gestions are proffered for continuing research Into evaluation 

problems. ^ 

■ W- 
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